Add Impersonate Service Account argument by wintermi · Pull Request #2015 · dataform-co/dataform

wintermi · 2025-09-11T06:43:03Z

This PR adds an --impersonate-service-account argument to the run and test commands, along with the required changes to allow for the impersonation of service accounts without the need to change ADC or call gcloud

This would resolve issue #2000 and would be an alternative to solution than PR #2001

Impersonation could then be achieved by executing:

dataform run --impersonate-service-account=<sSERVICE_ACCT_EMAIL>

google-cla · 2025-09-11T06:43:08Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

camilleAmaury · 2025-10-30T10:44:42Z

+1, this would enable to use impersonation in CI rather than giving the rights directly to the CI service account.
There is no way to workaround that currently.

kolina · 2025-11-11T11:28:51Z

/gcbrun

kolina · 2025-11-13T23:33:49Z

Sorry for the late review. A couple of things:

Integration tests are failing, can you take a look at fixing them? Now we have a guide of running them locally
Let's resolve conflicts

kolina · 2025-11-13T23:17:49Z

+        clientConfig.authClient = new Impersonated({
+          sourceClient: authClient,
+          targetPrincipal: this.bigQueryCredentials.impersonateServiceAccount,
+          targetScopes: ['https://www.googleapis.com/auth/cloud-platform']


EXTRA_GOOGLE_SCOPES?

Not sure what you would like done here?

I mean using EXTRA_GOOGLE_SCOPES here instead of hard-coding

kolina · 2026-01-02T21:13:06Z

@wintermi, in the current version tests are failing due to linter checks: output

You can check errors using this lint script.

wintermi · 2026-01-05T13:31:51Z

Resynced the PR with the latest commit

wintermi · 2026-01-05T14:36:31Z

@wintermi, in the current version tests are failing due to linter checks: output

You can check errors using this lint script.

Fixed the linter issues

wintermi · 2026-01-05T14:37:06Z

@kolina ready for retesting, thanks

kolina · 2026-01-07T12:34:01Z

/gcbrun

kolina · 2026-01-07T12:45:04Z

+        clientConfig.authClient = new Impersonated({
+          sourceClient: authClient,
+          targetPrincipal: this.bigQueryCredentials.impersonateServiceAccount,
+          targetScopes: ['https://www.googleapis.com/auth/cloud-platform']


I mean using EXTRA_GOOGLE_SCOPES here instead of hard-coding

wintermi · 2026-04-10T00:59:44Z

I removed the parse-duration dependency to avoid CVE-2025-25283
Moving to the parse-duration@2.1.3 seemed a far bigger change than necessary.

wintermi · 2026-04-10T01:09:29Z

I added a IMPERSONATION_GOOGLE_SCOPES as the EXTRA_GOOGLE_SCOPES only mention drive.

kolina · 2026-04-14T20:57:28Z

            getCredentialsPath(argv[projectDirOption.name], argv[credentialsOption.name])
          );
+          if (argv[impersonateServiceAccountOption.name]) {
+            (readCredentials as any).impersonateServiceAccount =


Can we extend dataform.IBigQuery with your new option to avoid dynamic casts breaking static typing?

kolina · 2026-04-14T21:04:15Z

  return `${value} ${units[i]}`;
 }

+const DURATION_UNITS_IN_MILLIS: { [unit: string]: number } = {


Can you please elaborate about the effort to avoid it and upgrade parse-duration dependency?

…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

Replace regex-based parseCliDuration with manual character parsing. Add build.sh to package CLI/core artifacts for hephaestus-worker-base. Fix malformed mkdirp integrity entry in yarn.lock.

kolina · 2026-06-25T11:46:45Z

/gcbrun

kolina

One of the tests started to time out for some reason

Step #1: //cli/api:dbadapters/bigquery_test                                       FAILED in 61.1s
Step #1: //testing:index_test                                                     PASSED in 0.9s
Step #1: //sqlx:lexer_test                                                        PASSED in 0.5s
Step #1: //sqlx:format_test                                                       PASSED in 0.8s
Step #1: //examples:examples_test                                                 PASSED in 3.2s
Step #1: //core:utils_test                                                        PASSED in 0.6s
Step #1: //core:main_test                                                         PASSED in 36.5s
Step #1: //core:jit_context_test                                                  PASSED in 0.7s
Step #1: //core:jit_compiler_test                                                 PASSED in 0.7s
Step #1: //core:compilers_test                                                    PASSED in 0.6s
Step #1: //core:actions/view_test                                                 PASSED in 8.9s
Step #1: //core:actions/test_test                                                 PASSED in 4.6s
Step #1: //core:actions/table_test                                                PASSED in 24.8s
Step #1: //core:actions/operation_test                                            PASSED in 7.9s
Step #1: //core:actions/notebook_test                                             PASSED in 5.0s
Step #1: //core:actions/index_test                                                PASSED in 5.4s
Step #1: //core:actions/incremental_table_test                                    PASSED in 29.1s
Step #1: //core:actions/filename_override_test                                    PASSED in 7.5s
Step #1: //core:actions/declaration_test                                          PASSED in 9.6s
Step #1: //core:actions/data_preparation_test                                     PASSED in 6.5s
Step #1: //core:actions/assertion_test                                            PASSED in 32.5s
Step #1: //common/strings:stringifier.spec                                        PASSED in 1.1s
Step #1: //common/protos:index_test                                               PASSED in 0.7s
Step #1: //common/protos:build_test                                               PASSED in 0.2s
Step #1: //common/promises:index.spec                                             PASSED in 2.9s
Step #1: //common/flags:build_test                                                PASSED in 0.3s
Step #1: //common/errors:errors.spec                                              PASSED in 0.9s
Step #1: //cli/api:utils_test                                                     PASSED in 0.8s
Step #1: //cli/api:execution_sql_test                                             PASSED in 0.7s
Step #1: //cli/api:commands/jit/rpc_test                                          PASSED in 0.7s
Step #1: //cli:util_test                                                          PASSED in 0.8s
Step #1: //cli:index_run_e2e_test                                                 PASSED in 62.8s
Step #1: //cli:index_project_test                                                 PASSED in 6.2s
Step #1: //cli:index_init_test                                                    PASSED in 4.3s
Step #1: //cli:index_help_test                                                    PASSED in 5.2s
Step #1: //cli:index_compile_test                                                 PASSED in 11.5s
Step #1: INFO: 
Step #1: INFO: Build completed, 1 test FAILED, 524 total actions
Step #1: INFO: 524 processes: 368 internal, 103 linux-sandbox, 1 local, 52 worker.
Step #1: INFO: Elapsed time: 123.997s, Critical Path: 99.81s
Step #1: INFO: Found 36 test targets...
Step #1: [523 / 524] 35 / 36 tests, 1 failed; Testing //cli:index_run_e2e_test; 63s linux-sandbox
Step #1: [523 / 524] 35 / 36 tests, 1 failed; Testing //cli:index_run_e2e_test; 61s linux-sandbox
Step #1: [523 / 524] 35 / 36 tests, 1 failed; Testing //cli:index_run_e2e_test; 51s linux-sandbox
Step #1: ================================================================================
Step #1: Tests failed.
Step #1: 
Step #1: BigQueryDbAdapter > setMetadata correctly maps column descriptions       PASSED
Step #1: BigQueryDbAdapter > setMetadata handles action without columns           PASSED
Step #1: 
Step #1:         at processTimers (node:internal/timers:541:7)
Step #1:         at listOnTimeout (node:internal/timers:605:17)
Step #1:         at Timeout.<anonymous> (testing/test.ts:85:22)
Step #1:     Error: Timed out (30000ms).
Step #1: 
Step #1:                                                                         TIMEOUT
Step #1: BigQueryDbAdapter > tables() without schema lists all datasets and tables
Step #1: 
Step #1:         at processTimers (node:internal/timers:541:7)
Step #1:         at listOnTimeout (node:internal/timers:605:17)
Step #1:         at Timeout.<anonymous> (testing/test.ts:85:22)
Step #1:     Error: Timed out (30000ms).
Step #1: 
Step #1: BigQueryDbAdapter > tables() with schema filters correctly              TIMEOUT

kolina · 2026-06-25T11:58:26Z

    projectId = projectId || credentials.projectId;
    if (!clients.has(projectId)) {
-      clients.set(
+      const clientConfig: any = {


BigQueryOptions instead of any?

kolina · 2026-06-25T17:01:50Z

+      const clientConfig: any = {
        projectId,
-        new BigQuery({
+        scopes: EXTRA_GOOGLE_SCOPES,


Are projectId and scopes here used by the auth library if you ovewrite authClient? If they're not used in this case, I'd only set them in else branch below

kolina · 2026-06-25T17:23:32Z

@@ -0,0 +1,12 @@
+#!/bin/bash


Non-intended diff?

wintermi requested a review from a team as a code owner September 11, 2025 06:43

wintermi requested review from Ceridan and removed request for a team September 11, 2025 06:43

Ceridan requested review from kolina and removed request for Ceridan November 13, 2025 09:15

kolina requested changes Nov 13, 2025

View reviewed changes

ashish10alex mentioned this pull request Dec 23, 2025

[Dataform CLI] Error when using service account impersonation through ADC #2056

Closed

wintermi requested a review from kolina January 7, 2026 05:35

kolina reviewed Jan 7, 2026

View reviewed changes

wintermi force-pushed the main branch from 763681d to 21fb05f Compare April 10, 2026 00:42

wintermi requested a review from kolina April 10, 2026 01:09

kolina requested changes Apr 14, 2026

View reviewed changes

wintermi added 7 commits June 17, 2026 15:23

Add the '--impersonate-service-account' argument to the 'run' and 'te…

cf3881c

…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

Make changes as requested

8c2e093

Make changes as requested

35a0e62

Make changes as requested

a96b601

Add the '--impersonate-service-account' argument to the 'run' and 'te…

2086be4

…st' commands and the required changes to allow for the impersonation of service accounts without the need to change ADC

Make changes as requested

3b6cd51

Make changes as requested

26b6528

wintermi added 2 commits June 17, 2026 15:27

fix(cli): remove parse-duration dependency

868c5a3

refactor(cli): dedupe bigquery impersonation scopes

25c4480

wintermi force-pushed the main branch from eb2b141 to 25c4480 Compare June 17, 2026 06:56

refactor(cli): rewrite duration parser and add packaging build script

3891fdb

Replace regex-based parseCliDuration with manual character parsing. Add build.sh to package CLI/core artifacts for hephaestus-worker-base. Fix malformed mkdirp integrity entry in yarn.lock.

kolina requested changes Jun 25, 2026

View reviewed changes

Uh oh!

Conversation

wintermi commented Sep 11, 2025

Uh oh!

google-cla Bot commented Sep 11, 2025

Uh oh!

camilleAmaury commented Oct 30, 2025

Uh oh!

kolina commented Nov 11, 2025

Uh oh!

kolina commented Nov 13, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolina commented Jan 2, 2026

Uh oh!

wintermi commented Jan 5, 2026

Uh oh!

wintermi commented Jan 5, 2026

Uh oh!

wintermi commented Jan 5, 2026

Uh oh!

kolina commented Jan 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wintermi commented Apr 10, 2026

Uh oh!

wintermi commented Apr 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kolina commented Jun 25, 2026

Uh oh!

kolina left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants